Privacy-preservation for Stochastic Gradient Descent Method

The traditional paradigm in machine learning has been that given a data set, the goal is to learn a target function or decision model (such as a classifier) from it. Many techniques in data mining and machine learning follow a gradient descent paradigm in the iterative process of discovering this target function or decision model. For instance, Linear regression can be resolved through a gradient descent method that iteratively minimizes the error of the target function. Other gradientdescent-based methods include Bayesian networks induction, genetic algorithms, and simulated annealing. These traditional algorithms assume free access to data, either at a centralized location or in federated form. Increasingly, privacy and security concerns restrict this access. Nowadays, data is often distributed among multiple parties: financial data is distributed across multiple banks and credit agencies; medical records are distributed across multiple hospitals and health care institutions and so on. secure multiparty computation and privacy preservation have attracted much attention in incorporating security into machine learning and data mining algorithms. A key issue in multiparty secure methods is to allow individual parties to preserve the privacy of its data, while contributing to the computation of a global result together with other parties. Many methods have been proposed to perform secure multiparty computation. For instance, the scalar product is a basic operations required in data mining. Recently, Li considered a privacy-preserving method [7] for Batch Gradient Descent (BGD) where each party learning a local classifier on its own data, and then gave two options on how to use the computed classifiers: all parties jointly compute the final classifier within a privacypreserving way; another is using their local classifier to jointly perform on-demand prediction whenever an unknown sample’s target value is required. The solution [7] proposed by Li is attractive, however, Batch Gradient Descent (BGD) method is costly for large scale data set problem and some randomized parameters in their method have not been declared in details. Another work [1] is a differentially private Stochastic Gradient Descent (SGD) algorithm for multiparty setting proposed by Rajkumar. Here, we propose a privacy-preserving solution based on Stochastic Gradient Descent (SGD) procedure by using the paillier cryptosystem. Our proposal achieves more strict privacy than [7], in our protocol, each party knows nothing but their own data during the update process. And it is a different solution by using cryptography to ensure privacy rather than random disruption in [1].